Experts Combination through Density Decomposition

This paper is concerned with an important issue in Statistics and Artiicial Intelligence, which is problem decomposition and experts (or predictors) combination. Decomposition methods usually adopt a divide-and-conquer strategy which decomposes the initial problem into simple sub-problems. The global expert is then obtained from some combination of the local experts. In the case of Hard decomposition, the local experts partition the input space without overlap, while in soft decomposition, experts overlap and may specialize on a common region of the input space. The overlap usually makes combination more powerful, since it introduces redundancy and smoothness in the experts outputs. On the other hand, there is a considerable literature on experts combination (independently of the decomposition) which consists in combining several global experts. Most of the studies deal with linear combination whose theory is better known and more tractable than the nonlin-ear case. There are several methods for linear combination which have been shown to improve performance by signiicantly reducing variancee3]]5]]1]. Although theoretical requirements for linear combination are not usually fulllled, because of the sample size and the statistical dependencies between individual experts, experimental results usually show improved performance over a single expert even when being chosen according to a cross-validation process. Actually, linear combination can be seen as a cross-validation in the functional (experts) space. The nal result is an average expert, while classical cross-validation (in parameters space) usually leads to some average performance and leaves the user choose a single expert according to some strategy (e.g. choose expert with the best performance). In this paper, we adopt a divide-and-combine strategy to solve a complex prediction task. The methodology consists in the following steps: 1. Assume the input density is a mixture of prior densities and estimate its parameters from the data. For our study we consider Gaussian distributions. 2. Decompose the global problem into a number of sub-problems according to a soft division process which is driven by the input density. The number of sub-problems is the number of Gaussians (clusters) in the mixture. 3. To every Gaussian (cluster), assign a local expert which is trained on the corresponding data (for which density is greater than a threshold, a small threshold allows more overlap between Gaussians of the mixture). 4. Combine the local experts. Determination of parameters of the combination is performed with two methods: i) Stacking, which consists in estimating weights which minimize some 1